GAAF: Searching Activation Functions for Binary Neural Networks Through Genetic Algorithm

Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices mobile systems. This is due to its significantly less computation storage demand, but at the of degraded performance. To close accuracy gap, this paper we propose add a complementary activation function (AF) ahead sign based binarization, rely on genetic algorithm (GA) automatically search for ideal AFs. These AFs can help extract extra information from input data forward pass, while allowing improved gradient approximation backward pass. Fifteen novel are identified through our GA-based search, most them performance (up 2.54% ImageNet) when testing different datasets network models. Interestingly, periodic functions key component discovered AFs, which rarely exist human designed Our method offers approach designing general application-specific BNN architecture. GAAF will be released GitHub.